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(54) Aflethod and system for selecting data for migration in a hierarchic data storage system using 
frequency distrilmtion tables 

(57) A hierarchic disk array data storage system (1 0) 
has multiple storage disks (12) that define a physical 
storage space (34) and a RAID management system 
(16) that maps the physk»i storage space (34) into two 
virtual storage spaces. A RAID-level virtual storage 
space (40) presents the physical storage space as mirror 
and parity RAI D areas where the mirror RAID areas store 
data according to RAID Level 1 and the parity RAID 
areas store data according to RAID Level 5. An applica- 
tion-level virtual storage space (50) presents the physical 
storage space as multiple virtual k)locks which can be 
assodated with min-or and parity RAID areas. The RAID 
management system (16) migrates the virtual biocks 
between the mirror and parity RAID areas to optimize 
peribrmance and reli^lity. To select virtual blocks lor 
migration from mirror to parity RAID areas, a frequency 
distnlxjtion table (60) is aeated to characterize the vir- 
tual blocks according to a predetermined distribution cri- 
teria. A virtual block is selected by segmenting the 
frequency distnlxition table (60) to identify virtual bh)cks. 
The application-level virtual storage space (50) is 
searched until a virtual block that satisfies the selection 
criterion is located. The located virtual block is then 
migrated to the parity RAID area. 
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Description 

FIELD OF THE INVENTION 

This inventbn relates to methods for selecting data 
for migratiori in a hierarchic data storage system using 
frequency distribution tattles. This invention also relates 
to a hierarchic data storage ^em which implements 
such frequency distrit3utlon tables. 

BACKGROUND OF THE INVFNTI QI^ 

Computer systems are constantly irrprcving in 
terms of speed, reliability, and processing capability. As 
a result, computers are Me to handle more complex and 
sophisticated applicatibns. As computers improve, per- 
formance demands placed on mass storage and 
irput^oulput (I/O) devices increase. There is a continuing 
need to design mass storage systems that keep pace in 
terms of performance with evolving computer systems. 

This invention particularly concerns the mass stor- 
age systems of the disk array typa Disk array data stor- 
age system have mi^ple storage disk drive devices 
whfoh are arranged and coordinated to form a single 
mass storage system. There are three primary design 
criteria for mass storage systems: cost, performance, 
and availability. It is most desirable to produce memory 
devices that have a low cost per megabyte, a high 
input/output performance, and high data availability. 
"Availability" is the ability to access clala stored in the 
storage system and the ability to Insure continued oper- 
ation in the event of some failure. Typfoally» data availa- 
bility is provided through the use of redundancy wherein 
data, or relationships among data, are stored in multiple 
locations. 

There are two common methods of storing redun- 
dant data According to the first or "minror" method, data 
is duplicated and stored in two separate areas of the stor- 
age system. For example, in a disk array, the identical 
data is provided on two separate disks in the disk array. 
Tbe mirror method has the advantages of high perform- 
ance and Ngh data availability due to the duplex storing 
technkiue. However, the miiror m^hod is also relatively 
expensive as it effectively doubles the cost of storing 
data. 

In the second or "parity" method, a portion of the 
storage area is usedtostore redundant data, but the size 
of the redundant storage area is less than the remaining 
storage space used to store the original data. For exam- 
pie, in a disk array having five disks, four disks might be 
used to store data with the fifth disk being dedfoated to 
storing redundant data. The parity method is advanta- 
geous because rt is less costly than the mirror method, 
but it also has lower performance and availability char- 
acteristics in comparison to the minror method. 



SUMMARY OF THE INVENTION 

This inventfon provkJes a data storage system that 
optimizes performance and reliability. The data storage 
5 system includes a disk array fiaving a plurality of storage 
disks and a disk array controller to coordinate data trans- 
fer to and from the disks. The storage disks define a 
physfoal storage space. The data storage system also 
includes a RAID management system operatively cou- 
10 pled to the disk array controller for mapping two virtual 
storage spaces into the physical storage space of the 
storage disks. A RAID-level virtual storage space 
presents the physical storage space as mirror and parity 
RAID areas that store data according to RAID Level 1 
15 (mirror redundancy) and RAID Level 5 (parity redun- 
dancy). An application-level virtual storage space 
presents the physical storage space as multiple virtual 
bfocks. The set of all virtual bfocks defines a population. 
The RAID management system migrates virtual blocks 
20 between the mirror and parity RAID areas so that data 
undergoes a change in redundancy from FkAID Level 1 
to RAID Level 5. 

The data storage system conducts migration t>ased 
Mpon one of two preferred policies: data access recency 
25 and data access frequency. Under the data access 
recency policy, data is migrated based upon how recentiy 
the data has been accessed by the user. Under the data 
access frequency policy, data is migrated based i4X>n 
how often the data is accessed by the user. OUerdata 
30 or less frequently accessed data is typically a candkiate 
for migratfon from the higher performance RAID Level 1 
to the lower performance RAID Level 5. 

The data storage system of this invention employs 
frequency dstribution tables to kJentify virtual blocks for 
35 migratfon. Preferably, one frequency distrilxition table is 
provided for each RAID level. The tables characterize 
data stored in the RAID areas according to a predeter- 
mined distribution criteria. The frequency disfritxition 
tables are bounded with a set number of intervals. A 
40 count is associated witti each inten/al to represent the 
number of virtual blocks that satisfy the distrfoution cri- 
teria for that interval. The total number of virtual blocks 
in each frequency disfribution table is the count of the 
population. 

45 The RAID management system selects the virtual 
bfocks for migration between a source RAID area arxi a 
target RAID area by segmenting the frequency disfribu- 
tion table of the source RAID area to identify a selection 
criterfon for virtual blocks. The application-level virtual 

50 storage space is searched only until a virtual block that 
satisfies the selection criterion is fr)und. The located vir- 
toal block is then migrated from the source RAID area to 
the target RAID area. 

The use of frequency disfribution tables is advanta- 

55 geous because the data storage system need not sort 
all virtual blocks to detect the oWest or least frequentiy 
used. Rather, the data storage system merely identifies 
a selection criterion for selecting candkiates for migra- 
tion, and uses tiie criterfon to select candidates from a 
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subset of the population having a sufficient number of 
candidates for migration. The process can be halted 
early upon locating enough virtual blocks to satisfy the 
storage request. Considerable efficiency is gained 
through this technique. g 

DESCRIPTION OF THE DRAWIMQR 

Preferred embodiments off the invention are 
described below with reference to the following accom- 10 
panying drawings depicting examples embodying the 
best mode for practidng the invention. 

Fig. 1 isadiagrammaticblockdiagramofadiskanray 
data storage system according to this invention. 

Fig. 2 is a diagrammatic illustration of storage space is 
on multiple storage disks and shows data storage 
according RAID Level 1. 

Fig. 3 is a diagrammatic illustration of storage space 
on multiple storage disks and shows data storage 
according RAID Level 5. 20 

Fig. 4 is a diagrammatic illustration of a memory 
mapping arrangement of this invention where twvo virtual 
storage spaces are mapped onto a physical storage 
space. 

Fig. 5 is a frequency distribution table according to 2s 
this invention. The frequency disfribution table is 
depkrted in Fig. 5 in its statistical bar graph representa- 
tion. 

Fig. 6 is the frequency distribution tat)le off Fig. 5, 
represented in an implementation format with pointers, so 
Rg. 6 demonstrates how the table is initially f iUed. 

Fig. 7 shows an updating process resulting from 
accessing an existing virtual bfock in the frequency dis- 
fribution table off Rg. 5. 

Fig. 8 shows the frequency distribution table of Rg. as 
5 before joinder of two intervals and the same frequency 
cGsfributiontablefollowingjolnd^oftwo intervals to Illus- 
trate a technique for generating a new interval within the 
bounded tabHe. 

Fig. 9 shows two frequency disfribution tables, one 40 
for RAID Level 1 and a second for RAID Level 5, where 
the RAID Level 1 table is the same as that shown in Rg. 
5. Rg. 9 demonstrates the effect on the tables resulting 
from migration of data between the two RAID levels. 

Fig. 10 is a f fow diagram of a mettiod for initializing 45 
frequency disfribution tables errpfoyed in a hierarchy 
data storage system according to this invention. 

Fig. 1 1 is a ffow diagram off using the frequency dis- 
tribution tables during a host access mode of operatfon 
according to this invention. 

Fig. 12 is a ffow diagram off using the frequency dis- 
fribution tables during a migration mode off operation 
according to this invention. 

DETAILFD DF SCRIPTIQN OF THE INVENTION 

This disctosure off ttie invention is submitted in fur- 
therance off the constitutional purposes off the U.S. Patent 



Laws "to promote the progress of science and useful 
arts". U.S. Constitution. Article 1, Section 8. 

Rg. 1 shows a data storage system 10 consfructed 
according to this invention. Preferably, data storage sys- 
tem lOisadisk array data storagesystemwhichindudes 
a hierarchic disk anray 1 1 having a plurality off storage 
disks 12, a disk array confroller 14 coupled to the disk 
array 1 1 to coordinate data fransfer to and from the stor- 
age disks 12. and a RAID management system 16. 

For purposes of this dsdosure, a "disk" is any non- 
volatile, randomly accessible, rewritable mass storage 
device which has the ability of detecting its own storage 
failures. It includes both rotating magnetic and optical 
disks and solid-state disks, or non-volatile electronic 
storage elements (such as PROMs. EPROMs, and EEP- 
ROMs). The term "disk array" is a colleclfon of disks, the 
hardware required to connect them to one or more ffiost 
computers, and management software used fr> cnnfrol 
the operation off the physfoal disks and present them as 
one or more virtual disks to the host operating environ- 
ment A "virtual disk" is an abstract entity realized in the 
disk array by the management software. 

The term "RAID" (Redundant An^^ off Independent 
Disks) means a disk array in whfoh part off the physfoal 
storage capacity is used to store redundant information 
atx)ut user data stored on the remainder off the storage 
capacity. The redundant information enables regenera- 
tion of user data in the event that one off the arra/s mem- 
ber disks or the access path to it foils. A more detailed 
discussfon off RAID systems is frMjnd in a book entrtied. 
The RAIDBook: A Source Book for RAID Technology, 
published June 9, 1993, tyy the RAID Advisory Board. 
Lino Lakes. Minnesota. 

Disk array confroller 14 is coupled to disk arre^ 1 1 
via one or more interfoce buses 1 3. such as a small com- 
puter system interface (SCSI), RAID management sys- 
tem 16 is operatively coupled to disk an-ay controller 14 
via an interface protocol 15. Data memory system 10 is 
also coupled to a host computer (not shown) via an I/O 
interlace bus 17. RAID management system 16 can be 
embodied asaseparate component or configured within 
disk array controller 14 or within the host computer to 
provide a data manager means for controlling disk stor- 
age and rellabiiity levels, and for transferring data among 
various reliability storage levels. These reliability storage 
levels are preferably mirror or parity redundancy levels 
asdescrit>ed below, but can also include a reliat)i&ty stor- 
age level with no redundancy at all. 

The disk anay confroller 14 is preferably imple- 
mented as a dual confroller consisting off disk army con- 
froller A 14a and disk ann^ controller B 14b. Dual 
controllers 14a and 14b enhance reliability by provkling 
continuous backup and redundancy in tiie event that one 
controller becomes inoperable. This invention can be 
practk^ed. however, witti a single confroller or ottier archi- 
tectures. 

The hierarchic disk an-ay 1 1 can be characterized as 
diffferent storage spaces, including its physical storage 
space and one or more virtual storage spaces. These 
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various views of storage are related through mapping 
technlquesw For exaiiple. the physical storage space of 
the disk array can be mapped intoavirtual storage space 
which delineates storage areas aocorcfing to the various 
data reliability levels. Some areas within the virtual stor- 
age space can be allocated tor a first reliability storage 
le/el, such as mirror or f^lD level 1 , and other areas can 
be allocated for a second reliability storage level, such 
as parity or RAID level 5. The vartous mapping tech- 
niques and virtual spaces concerning RAID levels are 
desaibed below in more detail. 

Data storage system 10 includes a memory map 
store 21 that provides for persistent storage of the virtual 
mapping information used to map different storage 
spaces onto one another. The memory map store is 
external to the disk array, and preferably resident in the 
disk anay controller 14. The memory mapping informa- 
tton can be continually or periodically updated by the 
controller or RAID management system as the various 
mapping configurations among the different views 
change. 

Preferably, the memory map store 21 is embodied 
as two non-volatile RAMs (Rarxtom Access Memory) 
21a and 21b which are located in respective controllers 
14a and 14b. An example non-volatile RAM (f^VRAM) is 
a battery-backed RAM. A battery-backed RAM uses 
energy from an independent t>attery source to maintain 
the data in the memory for a period of time in the event 
of power toss to the data storage system 10. One pre- 
ferred construction is a self-refreshing, battery-backed 
DRAM (Dynamic RAM). 

The dual NVRAMs 21a and 21b provide for redun- 
dant storage of the memory mapping information. The 
virtual mapping infonmation is dif)licated and stored in 
txyth NVRAMs 21a and 21b according to minor redun- 
dancy technkfues. In this manner, MVRAM 21a can be 
dedicated to storing the original mapping information 
and NVRAM 21b can be dedicated to storing the redun- 
dant mapping information. In an alternative construction, 
a mirrored memory map store can be configured using 
a single non-volatile RAM with suffkaent space to store 
thedataindMpHcala 

AssfiGwninFig. I.diskarray 11 has multiple storage 
disk drive devices 12. Example sizes of these storage 
disks are one to three Gigabytes. The storage disks can 
be independently connected or disconnected to 
mechanical bays that promde interfacing with SCSI bus 
13. In one implementation, the data storage system is 
designed with twelve active mechanical bays and four 
additional mechanical bays. Four SCSI buses are used 
to interface these bays with disk array controller 1 4 (i.e., 
one bus per 3-4 mechanical bays). If the active bays are 
fully loaded, the data storage system has an eoomple 
combined capacity of 12-36 Gigabytes. Disk array con- 
troller 14 recognizes storage disks 12 regardless into 
which t)ay they areplugged. The data storage system 10 
is designed to permit "hot plug" of additional disks into 
available mechanical bays in the disk array while the disk 
array is in opeiBtion. 



The storage disks 12 in disk array 1 1 can be con- 
ceptualized, fbr purposes of explanation, as being 
arranged in a mirror group 1 8 of multiple disks 20 and a 
parity group 22 of multiple disks 24. Min-or group 18 rep- 

5 resents a first menx>ry location or RAID area of the disk 
array which stores data according to a first or mirror 
redundancy level. Thte minrcx redundancy level is also 
considered a RAID Level 1. RAID Level 1. or disk mirror- 
ing, offers the highest data reliat)ility by providing one- 

10 to-one protection in that every bit of data is duplicated 
and stored within the data storage system. The minor 
redundancy is diagrammatically represented by the 
three pairs of disks 20 in Rg. 1. Original data can be 
stored on a first set of disks 26 while diplicative, redun- 

15 dant data is stored on the paired second set of disks 28. 
Rg. 2 illustrates the storage of data according to 
RAID L.evel 1 in more detail. The vertk^l columns repre- 
sent indivkiual disks, of which disks 0, 1, 2. and 3 are 
illustrated. The physical storage space contained in this 

20 disk array of four disks can be configured into multiple 
stripes, as represented by tiie horizontal rows. A "stripe" 
extends across the storage disks and is comprised of 
numerous, equal sized segments of storage space 
where one segment is associated with each disk in the 

25 array. That is, a segment is the portion of a stripe tfiat 
resides on a single disk. Each stripe holds a predeter- 
mined amount of data which is distrilxjted across the 
storage disks. Some segments of a stripe are used for 
original data while other segments are used for redun- 

30 dantdata. 

In this exanple of mirror redundancy (RAID Level 
1), data stored on disk 0 in segment 0 of stripe 0 is dupli- 
cated and stored on disk 1 in segment 0' of stripe 0. Sim- 
ilariy, data stored on disk 2 in segment 5 of stripe 2 is 

35 mirrored intosegmentS' of stripe2 ondisk3. In this man- 
ner, each piece of data is duplicated arvJ stored on tiie 
disks. The redundancy layout of Fig. 2 is provided for 
explanation purposes. The redundant data need not be 
placed neatly in the same stripe as is shown. For exam- 

40 pie, data stored on disk 0 in segment 2 of stripe 1 could 
be duplicated and placed on disk 3 in segment T of stripe 
S. 

With reference again to Rg. 1, tiie parity group 22 
ofdisks24representasecond memory location or RAID 
45 area in whkih data is stored according to a second redun- 
dancy level, such as RAID Level 5. In tiiis explanatory 
illustration of six disks, original dafta is stored on the five 
disks 30 and redundant "parity" data is stored on the sixth 
disk 32. 

so Rg. 3 shows a parity RAI D area layout in more detail. 
Similar to the mirror RAID area layout of Fig. 2. the phys- 
icsA storage space of disks 0, 1 , 2. 3 can be configured 
into multiple equal sized stripes. In this illustrated exam- 
ple, data is stored according to RAID Level 5 and the 

55 redundant data stored in the segments is referenced by 
letter P. The redundant P segments store the parity of 
the other segments in the stripe. For example, in stripe 
0. the redundant P segment on disk 3 stores the par'rty 
of disks 0. 1 . and 2. The parity fbr each stripe is computed 
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by some function, such as an exclusive OR function 
which is represented by the symbol The parities for 
the first four stripes (with the subsaipt numeral repre- 
senting the corresponding stripe) are as follows: 

5 

Po = SegmentO © Segment 1 a Segment 2 
= Disk0eDi8k1 eDisk2 

Pi = Segment 3 ©Segment 4 ©Segment 5 

= DiskO©Disk1 ©Disk3 io 

P2 = Segment 6 © Segment 7 © Segment 8 
= Disk 0 © Disk 2 © Disk 3 

P3 = Segment 9 ©Segment 10 ©Segment 11 is 
= Dlsk1 ©Disk2©Disk3 

Parity redundancy allows regeneration of data which 
becomes unavailable on one of the disks. For exanple. 
if the data in segment 5 becomes unavailable, its con- 20 
tents can be ascertained from segments 3 and 4 and the 
parity data in segment P. Parity storage is less expensive 
than minor storage, but is also less reliable and has a 
lower performance. 

The disk anangement of Fig. 1 is provided for con- 2s 
ceptual purposes. In practfoe. the disk anay 11 would 
siniply have a plurality of disks 12 vvhich are capable of 
storing data according to minror and parity redundancy 
Among the available storage space provided by all disks 
12, a portion of that storage space wouki be alkx;ated 30 
for minor redundancy and another portfon would be alk>- 
cated for parity redundancy Preferably, disks 12 are con- 
figured to contain plural, equal sized storage regions 
(referenced as numeral 35 in Rg. 4). wherein individual 
regpons have multple segments. The regions are as 
grouped together to form RAID are^ in one virtual view 
of thestorage space. Additionally. another(hoBt-<lefined) 
view of storage space is presented to the user or host so 
that the RAID areas and data redundancy storing tech- 
niques are transparent to the user or host These lea- 40 
tures are discussed beksw in more deiafl with reference 
to Fig. 4. 

Data storage system 10 manages the "migration* of 
data between minror and parity storage schemes. The 
management of both types of redundancy iscoordinated 4s 
by RAID management system 16 (Rg. 1). RAID man- 
agement system 16 manages the two different types off 
RAID areas in the disk anray as a menmry hierarchy with 
the mirror RAID areas acting similar to a cache for the 
parity RAID areas. RAID management system 16 shifts, so 
organizes, and othenwise manages the data between the 
mirror and parity RAID areas in accordance with a 
defined performance protocol. The process of moving 
data between tiie mirror and parity RAID areas is 
referred to as "migration". 55 

Data storage system 1 0 tries to place the more per- 
formance-critical data in the minror RAID areas since this 
affoids the highest performance and reliability. The per- 
fonriance protocols implemented tiy RAID management 



system 16 includes one of two preferred migration poli- 
cies. According to the first migration polk;y. known as 
"access frequency", the most frequently accessed data 
on the hierarchic disk array is maintained in tfie mirror 
RAID area 18. Less frequentiy accessed data is main- 
tained in the parity RAID area 22. According to a second 
migration policy, known as "access recency", the most 
recentty accessed data is maintained in the minror RAID 
area 18 while the less recently accessed data is stored 
in parity RAID area 22. Other performance protocols may 
be employed. Ideally, such protocols are defined based 
upon the specific computer appficafion and tiie needs of 
the user. 

Additionally, the RAID management system 1 6 auto- 
matically "tunes" the storage resources of a data storage 
system according to a function of two parameters: size 
of tile physical storage capacity arxl size c* the present 
amount of user data being stored in the data storage sys- 
tem. Initially, all data is stored in mirror RAID areas 
because this affords the highest performance and relia- 
bility. As more data is added to the data storage system, 
ttie data is migrated between min'or RAID areas and par- 
ity RAID areas to optimize performance and reliability 
As the data storage system approaches full capacity, 
more and more data is migrated to parity RAID areas in 
an effort to meet all demands by the user while still pro- 
viding reliability through redundancy. Accordingly, the 
data storage system of this invention affords maximum 
flexibility and adaptation. It does not require the user to 
select a spedffo storage regime, but instead can adapt 
to any demand placed on it by the user. 

Rg. 4 illustrates a memory mapping of the available 
storage spaceofdatastoragesystem lOasmultipletiers 
of mapped virtual storage space. Each vertfoally elon- 
gated rectangle in the diagram represents a view of the 
physfoal storage space. In this diagram, physical storage 
space 34 is referenced by two virtual storage views 40 
and 50. Physical storage space 34 is represented by four 
disks (such as disks 12 in Fig. 1) referenced by numerals 
0. 1, 2. and 3. The four rectangles associated with the 
disks represent a view of the physfoal storage space 
wherein disks 1 , 2, arxJ 3 have approximately equal stor- 
age capacity, and disk 0 has slightly less storage capac- 
ity. Example storage capacities for such disks are 1-3 
Gigabytes. The storage space 34 is partitioned into 
areasAO. A1. A2.ete. Individual areas contain numerous 
stripes, such as stripes 0-Q in area AO. Individual areas 
also contain numerous regions 35. Regions 35 prefera- 
bly consist of a selected number of uniform sized seg- 
ments on everystoragedisksothatthe regions are equal 
in size aaoss the entire disk array. An example size of 
one region 35 is one Megabyte. 

The storage space of the disks are mapped into a 
first, intermediate. RAID-level virtual view 40 of the phys- 
ical storage space 34. This first virtual view is conceptu- 
ally a set of RAID areas which can be mapped to a 
second applicatfon view that represents a contiguously 
addressable storage space. The ptiysfoal configuration 



5 



9 



EP0709765 A2 



10 



and RAID views of the storage space are hidden from 
the application view. 

The RAID area storage space 40 is the view of stor- 
age that identifies the mirfDr and parity storage space. 
For instance, a RAID area 42 may represent a mirror s 
BA\D area of M allocation l^locks 43 while RAID area 44 
may represent a parity RAID area of N allocation blocks 
45. The allocation tHocAss 43 and 45 are preferably equal 
sized, with an example size being 64 Kiloft>yte8. These 
RAID areas relate to corresponding physical areas AO, io 
A1 A2. ete., on the pfiysical storage space 34. As an 
exanrple, sixteen 64K allocation blocks 43 or 45 at tfie 
RAID virtual view can be mapped onto a single 1 M region 
35. 

The minror and parity RAID areas may or may not is 
consume the entire storage space 34 of ttie disk array 
Accordingly during certain applications, there may be 
unused and undesignated storage space that does not 
correspond to a particular RAID area. However, such 
storage space can be converted into a minor or parity so 
RAIDarea. It is also noted thatthe RAID are^ are shown 
as being mapped into contiguous areas on the disk array, 
where each region associated with a RAID area is 
located at tiie same physical address on each storage 
disk. The RAID areas may alternatively be mapped Into 2s 
non-contiguous areas on the disk array as well. 

The storage space available in the RAID areas is 
mapped into a second, front end. application-level virtual 
view 50 which is a view off storage as defined by and pre- 
sented to the user or host application program. When 30 
viewed by the user or host application pro^m, the appli- 
cation-level virtoal view 50 can represent a single large 
storage capacity indicati ve of the availatMe storage space 
on storage disks 12. Virtual storage space 50 presents 
a view of a linear set of equal sized storage virtual blocks as 
52 and 53, referenced indivkfuaHy as 0, 1, 2,,.. J-l, j, 

J+1 L-1. U U1 ete. Virtual btocks 52 and 53 are 

preferably the same size as the allocation bkx:ks in RAID 
area 40, with an example size being 64 Kilobytes. The 
virtual block storage space 50 is represented by a table 40 
of references or pointers (as represented by arrows 54) 
to altocation btocks in the view presented by RAID areas 
40. Virtual blocks^ and 53 at the application virtual view 
50 are therefore assodated with allocation blocks 43 and 
45 at the RAI D virtual view 40 via the pointers maintained 4s 
in tiie virtual block table. There are at least two types of 
RAID areas that can be referenced from the virtual bk)ck 
table: minor and parity 

The RAID management system 16 can dynamically 
alter tiie confi^rab'on of the RAID areas over the phys- so 
ical storage space. The number of RAID areas for each 
type may be increased or decreased depending upon the 
amount off user data being stored in the system and ttie 
size of ttie physical disk storage space. As a result, tiie 
mapping of tiie RAID areas in ttie RAID-level virtual view ss 
40 onto ttie disks and the mapping of ttie front end virtual 
view 50 to RAID view 40 are generally in a state of 
change. The memory map store in MVRAMs 21a and 
21b (Rg. 1) maintoins ttie current mapping infbnnation 



used by RAID management system 16 to map the RAID 
areas onto ttie disks, as well as ttie infbnmation enployed 
to m^ between ttie two virtual views. As the RAID marv 
agement system dynamically alters ttie RAID level map- 
pings, it also updates the mapping information in the 
memory map store to reflect the alterations. 

The migration operation of memory system 10 will 
now be descrftied with reference to Rgs. 1 and 4. 

For purposes off continuing explanation, virtual 
bkx:ks 53 of the application-level virtual storage space 
50 reference associated allocation blocks 45 in parity 
RAID area 44 stored in area A1 of physical storage qpace 
34. Such virtual bk)cks 53 are referred to as "parity virtual 
bkx^cs" while ttie associated alk>cation btocks 45 are 
referred to ^ "parity allocation blocks". Similarly, virtual 
bkxte 52 reference associated allocation blocks 43 in 
mirror RAID area 42 stored in area AO of physical storage 
space 34. Such virtual bkx^ 52 are referred to herein 
as "mirror virtoal blocks" while ttie assodated altocation 
btocks 43 are referred to as "min-or allocation blocks". 

In general, to migrate data from one RAID area to 
anottier. a virtual btock assodated witti an allocation 
btock of a first RAID level type (such as minror or Level 
1) is selected. Then, an unused alk)cation btock repre- 
senting a second RAID level type (such as parity or Level 
5) is tocated. If an unused allocation block cannot be 
located, one is created. Data is next transferred from the 
allocation btock prevtously associated witti the selected 
virtoal btock to ttie unused allocatton block which causes 
ttie data to undergo a redundancy level change. For 
example, data once stored according to mirror redun- 
dancy wouM now be stored according to parity redun- 
dancy, or vice versa. As a final step, ttie mapping 54 of 
ttie application-level virtual storage space 50 to the 
RAID-level virtual storage space 40 is modih'ed and 
updated to reflect ttie shift of data. The selected virtual 
btock that was formerly associated witti an allocation 
block of the first RAID level type now references via an 
updated pointer an allocation block of the second RAID 
level type which contains ttie migrated data. Any map- 
ping change occunring during ttiis transaction wouM be 
updated in memory map store 21 . 

The continuing discussion provkies a more detailed 
explanation of migrating data between mirror and parity 
storage areas according to preferred methods and 
sequences of this invention. To migrate data from parity 
to mirror storage, ttie foltowing sequence is emplc^ed: 

1. The RAID management system locates an 
unused mirror aliocatbn block 43 in a mirror RAID 
area 42. 

2. If none can be found, the RAID management sys- 
tem creates a mirror allocation block (discussed 
below). 

3. The RAID management system suspends new 
storage requests to ttie virtual btock to be migrated. 

4. The RAID management system waits until all 
active data storage requests to the virtual block are 
completed. 
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5. TTiedata from the parity allocation block 45 asso- 
ciated with virtual block 53 Is read into a temporary 
memory buffer. 

6. Tlie data is then written to the mirror allocation 
tilock 43 chosen in step 2. s 

7. The virtual blocktable is modified to reference the 
new location of the data in the min-or allocation block 
43. 

8. The suspended storage requests are resumed. 
According to the akxyve procedure, a virtual block 53 io 
in the application-level virtual view 50 migrated from 
parity to mirror storage. Relative to the intermediate 
virtual view 40. data has migrated from a parity atto- 
cation tHock 45 in parity RAID area 44 to a mtrror 
allocation block 43 in minor RAID area 42. In the 75 
physical storage space data has nmed from area 

Al to area AO. 

If an unused mirror ailocatxHi tHock cannot be 
located (step 1 above), the RAID management system 20 
tries the following preferred sequence of three tech- 
niques. First, the RAID management system will try to 
locate an unused (and thus undesignated) RAID area, 
which can be converted to a mnror RAID area witfiout 
vk)latingthesystemthresholdof unused RAID-level stor- 25 
age that is needed to guarantee that migration can 
always proceed. If this fails and the system has more 
than the reserved amount of unused RAID-level storage, 
the system migrates data within parity storage to collect 
unused parity allocation bkxAs into unused RAID areas. 30 
If this rn^tion yields an unused RAID area that can be 
converted to a mirror RAID area as atx)ve. then the sys- 
tem converts it to a mirror RAID area. Othenwise. the sys- 
tem alternately migrates data from mirror to parity 
storage, packs minor storage, and converts unused as 
RAID-level storage to parity until the system increases 
unused RAID-level storage suff K^iently for the kx;ation of 
an unused mirror allocation bk)ck or a conversron of an 
unused RAID area to a mirror RAID area. Since mirror 
allocatton blocks occupy more physical storage space 40 
tfian parity allocation btocks. this last technique will result 
in a n^ Increase in the amount of unused RAID-Jevel 
storage. 

The creation/conversion protocol used to locate and 
estak)lish unused mirror allocation blocks is advanta- 45 
geous t)ecause it permits the RAID management system 
to selectively adjust the mmory altocalton between par- 
ity and mirror areas according to the amount of user data 
and the size of f^iysical storage spaca As data usage 
and storage capacity vary, the RAID nianagement sys- 
tem employs one or more of the above three techniques 
to rvtaximize the amount of data hekl in mirror storage. 

The RAID management system attempts to avokl 
the situation in which a storage request must wait for the 
space-making sequence to yield an unused mirror allo- 
cation t)lock by creating unused RAID areas during idle 
time. However, in some situations, storage requests may 
t>e suspended during the space-making sequenca The 
RAID management system configures the virtual btock 



storage space in such a way that the virtoal space will 
be smaller than the RAID view. This ensures tfiat a free 
space equal to at least one RAID area is set askfe for 
miration or other purposes. In this manner, the 
sequence of technk^ues will always yield an unused mir- 
ror allocation bk)ck. 

To migrate data from minror to parity storage, the fol- 
lowing sequence is employed: 

1. The RAID management system chooses a virtual 
blockfrom 52 to migrate from mirror to parity storage 
according to a migratton polk;y such as access 
recerx:y or access frequency. 

2. The RAID management system k}cates an 
unused parity alkKatton btock 45 in a parity RAID 
area 44. 

3. If such a block cannot be found, space reserved 
for migration is converted to a parity RAID area 
according to the atx)ve described aeation tech- 
niques. 

4. New storage requests to the virtoal block to be 
migrated are suspended. 

5. The RAID management system waite until all 
active storage requests to the virtoal block are com- 
pleted. 

6. Data is read from the mirror allocation btock 43 
associated with virtoal block 52 into a temporary 
memory t)uffer. 

7. The data is written to the chosen parity allocation 
block AS. 

8. Thevirtoal t)lock table is modified to reference the 
new location of the date in parity alkx:ation YAock 45. 

9. Data requests to tiie virtual block are resumed. 
The above two nine-step sequeru^es provkle exam- 
ples of how the memory system of this inventton can 
operate to migrate data t>6twem two different levels 
of redundancy. 

Data is migrated between parity and minor RAID 
areas using one of the two preferred migration policies 
of date access recency and data access frequency In 
general operation, the data storage system of this inven- 
tion wants to migrate the oldest or least frequentiy 
accessed virtual bk>cks from min^or to parity storage. 
Contrarily. it might be usetol to migrate the newest or 
most frequentiy accessed virtoal blocks from parity to 
mirror storage. One dl the tasks of the data storage sys- 
tem is to select the appropriate virtoal btocks for migra- 
tion. 

According to this invention, the virtoal blocks are 
tagged with a monofeonically increasing mark to assist in 
tiie selection process. K tiie migration policy is access 
recency, the virtual blocks are tagged with a time starrp 
indicating when the data was stored therein. The time 
stamp is preferably generated by a counter in the disk 
array controller 14 which outputs an inaemental tick 
about every minute (in one implementation, every 65.5 
seconds). The counter thereby forms suitable means for 
marking such storage bfocks. although other maridng 
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means may alternatively be employed. For access fre- 
quency, the virtual blocks are tagged wHh a frequency 
stamp indicating the number of times the data therein 
has been accessed. The frequency stamp Is an integer 
value that is increased by one with each access of the 
corresponding virtual block. 

One straightftKward technique for locating the 
appropriate virtual blocks for data migration is to sort the 
entire second virtual storage space 50 by time or fre- 
quency stamp. The sorting would yiekJ the oldest/young- 
est or least frequently/hiost frequently accessed virtual 
bk)Gk5 for migration between parity and mirror RAID 
areas. This technique is less preferred, however, 
t>ecause it takes considerable memory resource and 
processing time to sort all of the virtual blocks. The short- 
comings of the sorting technique t>ecome more pro- 
nounced as the size of user data increases. The 
processing time is a non-linear function of the msrber of 
virtual bbcks to be sorted. As the number of bkxAs 
increases, the processing time becomes kKiger. 

A second and more preferred technique of tiiis 
invention is to statistically select suitable virtual blocks 
without resorting to a full sort According to this tech- 
nique, one or mofe frequency distritxition tables are 
established by the RAID management system 1 6 to sta- 
tistically characterize the virtual blocks in temns of their 
time or frequency stamps. The frequency distritxition 
table(s) can then be processed to Identify a certain 
stamp threshoM that segments the oMer/younger or less 
frequentlyAnore frequently accessed virtual bk)cks from 
the remaining population of virtual blocks. Once a seg- 
ment stanrp tiireshokJ is identified, the application-level 
virtual storage space can be quickly searched until a suf- 
ficient number of virtual blocks within the segment have 
k>een found. The search effort is then halted. The data in 
the located virtual blocks is manipulated to carry out the 
desired function, such as migration to other allocation 
blocks. The statistical selection technique optimizes both 
memory usage and processing tinrte and is therefore pre- 
ferred over the full sorting technkjue. 

Rgs. 5-12 demonstrate the various uses^ creation, 
and maintenance of frequency distribution tables in hier- 
archic data storage systems according to this invention. 
These aspects and corresponding novel mettiods will be 
explained with reference to the iloM diagram of Rgs. 10- 
12. Preferably, the metiiods described herein are imple- 
mented as firmware in RAID management system 16. 

Rg. 10 illustrates an initialization process. The first 
two steps 100 and 102 establish the RAID environment 
for the hierarchic data storage system. At step 100. RAID 
management system 16 maps the physical storage 
space provkied by disk array 12 into RAID-level virtual 
storage space 40. The RAID-level virtual storage space 
presents the physical storage space as multiple RAID 
areas that store data according to various RAID levels, 
such as RAID Lev^ 1 and RAID Level 5. At step 1 02. the 
RAID management system tnsaps the RAID-level virtual 
storage space into applk»tion-level virtual storage 
space wfuch presents the physical storage space as mul- 



tiple virtual bfocks. The virtual blocks are associated with 
corresponding RAID areas. Data is stored in the virtual 
bfocks according to the RAID level of the associated 
RAID area. 

5 At step 1 04, the RAID management system creates 
one or more frequency distritxjtton tables. Preferably, 
one frequency distribution table is created for each type 
of RAID area. The frequency distribution tat)les are pref- 
erably formed in a volatile RAM in the disk array controller 

10 14 and rebuilt after each system initialization. Alterna- 
tively, the distribution tatAe can be reskient in tfie 
NVRAMsof memory mapstore 21 indiskarrayoontroller 
14. The NVRAM implementation is preferred if it is fore- 
seen that rdxuMing the table after each system initiali- 

15 zation is too time consuming. Aspects of these tables are 
described in more detail with reference to Rgs. 5 and 6. 

Rg. 5 illustrates a frequency c£stribution tat)le 60 in 
a bar graph format and Fig. 6 illustrates the same table 
in a memory format conceptualizatfon. Frequency distri- 

20 tHJtion table 60 statistically characterizes the data stored 
on the data storage system according to a predeter- 
mined distritxition criteria. Two preferred distrfoution cri- 
teria are data access recency or data access frequency. 
The table makes the characterizatk>n at the virtual Uodc 

25 level in second virtual storage space 50. Thus, the tat>le 
contains statistical information atxxit the virtual blocks 
t>elng used to store user data. 

Frequency distribution tak)le 60 is preferat)ly 
txxjnded wHh a fixed number of intervals An example 

30 nunrter of intervals is ten, as shown in Rgs. 5and6. The 
intervals are labeled generically as 1-10. but such inter- 
vals actually consist of a range of values from a begin- 
ning boundary value to an ending fcwundary valua If the 
distribution criteria is access recency, the intervals are 

35 temporal in scope where each Interval covers a range of 
lime stamps associated with the virtual btodcs. An exam- 
ple interval consists of 1024 time stamps which, at 65.5 
seconds per stamp, covers all of the virtual k)locks used 
in a time period of approximately one day. If the distritXJ- 

40 tion criteria is access frequency, the intervals relate to 
firequency values where each interval covers a range of 
frequency stampsassoctated with the virtual t>focks. The 
various inten/als and counts under the access frequency 
regime depend upon tiow actively the user accesses its 

45 data. 

The table contains the number of virtual blocks that 
OGCurwithin each inten/al. This isrepresented by thever- 
tical bars and associated counts In Fig. 5. For exanple. 
thirty-two virtual blocks have a time or frequency stamp 

50 that falls witiiin interval 1 ; forty-eigfit virtual blocks have 
a time or frequency stamp tiiat falls writhin intenml 2: and 
so on. The table also tabulates a cumulative frequency 
or total number of virtual bfocks to define the entire pop- 
ulation. In this case, the total population consists of 566 

55 virtual blocks. 

In Fig. 6, frequency cfistribution table 60 contains a 
first fieW 62 which stores the total population of 566. The 
table also contains a population queue 64. a free queue 
66, and ten Intend fields 68. Population queue 64 main- 
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tains the interval elements for the current population 
whereas free queue 66 tracks the unused interval ele- 
ments. Each interval field 68 contains the count for the 
associated inten/al Q.e., thirty-two for interval 1. forty- 
ei^ for interval 2, and so on), the beginning txHindary 5 
value, and the size of the inten^l. From the beginning 
boundary value and the interval size, the ending bound- , 
ary value can be readily computed. 

The frequency distrikxjtion table Is aeated during 
system initialization by scanning the virtual k)locks con- 10 
tained in the application-level virtual storage space. In 
the beginning, population queue 64 is empty and free 
queue 66 contains ten unused interval elements. The 
cowit and t>eginning boundary value lor all ten interval 
fields 68 default to zero The size for each interval field is 
is initialized to a default size, such as 1024. The taksle is 
then statistically constructed t>ased iqson the stamps, 
such as the time or frequency starrps, of the virtual 
blocks (step 106 in Rg. 9). The number of virtual blocks 
satisfying each interval is counted and input to the appro- 20 
priate interval field 68. 

The stamp for each virtual block is sequentially 
checked (as indk»ted by the flow arrows in Rg. 6) 
against the interval f iekJs 68 until a match occurs (step 
107 in Rg. 9). For example, assume thatthe virtual block 2s 
had a stamp that fell within the interval in the thiid interval 
field. The virtual block stamp woM first fc>e evaluated 
against the interval in the first interval field. Since the vir- 
tual block falls outskie this inten/al fieki, it is next com- 
pared to the second interval f ieM. It is noted that e^h 30 
inten/al f iekJ contains pointers to direct analysis to the 
address for the next interval field. The process continues 
f ieM by f iekl until the virtual block stanp is compared and 
found to fit in the appropriate interval f ieM (which in this 
case, is the third interoal fieU), whereupon the count for 3S 
that interval fieki in incremented (step 108 in Rg. 9). It is 
noted that mused virtual blocks have no stamp associ- 
ated with them, and thus are not represented in the table. 
By maintaining top and last intenal pointer values, the 
population queue 64 can readily find and point to the top 40 
and last interval fieUs 68. 

As virtual btocks are scanned and the statistical 
information extracted, populatfon queue 64 k>egins to fill 
and free queue 66 bogms to empty. When the tabUe con- 
tains a count in each interval, the free queue 66 will t>e 45 
empty (as denoted by the n\i\ or ground symbols in the 
free queue). It is noted that when two or more tables are 
desired (as described below in more detail), they can be 
created simultaneously during the same scanning step. 

Once the frequency disfritxjtion tables are estab- so 
lished, they are ready for use The f fow diagrams of Rgs^ 
1 1 and 12demonstratetwopreferred modes of operatfon 
involving the frequency distribution tables, although 
other operations using the frequency disfribution tables 
are likely. The first or host access nrxxle of operation is ss 
to update the tables as the user accesses existing or new 
virtual blocks. The second or migration mode of opera- 
tion concerns segmenting the tables to identify virtual 



btocks for use in migration between RAID areas and 
updating the frequency disfribution tables. 

Rg. 11 shows the host access mode of operation 
wherein the tables are continually updated as ecisting 
and new virtoal blocks are used to store user data. At 
step 109, it is first determined whether the user is 
accessing an existing virtual t>lock or is creating a new 
virtoal block. If the user is accessing an existing virtual 
bfock (i.e., the "no" Ixanch from step 1 09), a new stanrp 
is applied to the easting virtual block. TTie frequency dis- 
fribution tables are i^xlated to reflect tha new stamp. This 
is done by decrementing the count of the interval con- 
taining the old stamp (step 110) and then inaementing 
tiie count of the interval containing the new stamp follow- 
ing step 111. 

Rg. 7 illustrates tiie process of updating a tat)le 
when an existing virtual bfock is accessed. Here, a virtoal 
block having a time stamp in ttie range of interval 6 is 
accessed. A new, most recent time stamp is then tagged 
to the virtoal block. One significant advantage to having 
a monotonically increasing stamp is that the system can 
easily locate and handle the most recently/frequentty 
accessed and least recently/frequently accessed virtoal 
bfocks because these btocks are focated in the first and 
last intervals. The most recent time stamp therefore 
expectedly falls witiiin the range of interval 10. The fre- 
quency disfribution table 60 is updated by decrementing 
the coust of interval 6 by one (80 to 79) and incrementing 
the count of interval 10 by one (76 to 77). 

With reference to Rg. 11, if the virtual block is new 
(i e.. the "yes" branch of step 109). analysis continues at 
decision step 111 in Rg. 11 to determine whether the 
new virtoal bfock fits an existing interval within a table. 
Given the monotonic natore of stamping virtoal blocks, 
the virtual bfock is simply compared to the last inten/al 
of the taUa If tiie new virtual bfock fits the last interval 
(i e., the "yes" path from decisional step 1 1 1), the count 
associated wiUi that last interval is incremented (step 
112). For instance, if a virtual block has a time stamp 
falling in the range of interval 10, the count of interval 10 
is incremented from 76 to 77. On the other hand, if tiie 
virtoal block has a stamp that does not satisfy any of the 
last intervals (i.e.. the "no" path from decisional step 
111). the RAID managemerrt system quickly checks if 
any intervals are free (step 1 13). This situation occurs 
when less ttian tiie set numk)er of intervals (e.g.. ten inter- 
vals in the continuing example) have been used to 
describe tiie frequency disfribution of the last virtoal 
blocks. If an interval is free (i.e.. tiie "yes" branch from 
step 1 13), tiie new last interval is defined at step 1 16 to 
contain tiie new virtual bfock. 

When no free intervals exist (i.e.. the "no" branch 
from step 1 1 3), a new interval must t>e created to accom- 
modate the increasing distritxition range. Creating a new 
interval requires some adjustment to the frequency dis- 
fribution tat)le t)ecause it is bounded with a fixed nunri>er 
of intervals. The RAID management system does not 
merely add a new inten/al. Instead, the RAID manage- 
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ment system first mates room within the table to accom- 
modate the new interval. 

Rg. 8 Illustrates the process of generating a new 
interval. The RAID management system first combines 
adjacent intervals to form a new joint interval (step 114 s 
in Rg. 1 1). The new joint interval has a count that equals 
the sum of the counts of the two intervals that are joined. 
Preferal)ly, the adjacent two intervals with the smallest 
combined count are selected. In Rg. 8, intervals 2 and 
3 are joined to form a new interval 2 in table W. The /o 
count of the new interval 2 is 76 (i.e.. 48 + 28 = 76). 

This process frees up space in the frequency distri- 
bution table for definition of a new last interval (step 116 
in Rg. 11). such as new last interval 10. The count for 
the new last interval is then incremented to reflect the is 
new virtual block (step 118). In table 60'. the new last 
interval 10 has a count of one to signify statistically that 
the new virtual block fits that inten/al and did not fit the 
previous interval& Through the joining process, the fre- 
quency distribution tabfo remains bounded and easy to 20 
handle, and the statistical information therein is accu- 
rately maintained. 

As noted above. In the prefenred implementation of 
this invention, one frequency distribution tafckle is oon- 
staicted for each RAID level. When the hierarchic disk 25 
array uses RAID Levels 1 and 5, a first frequency distri- 
bution table is aeated for virtual blocks associated with 
min-or RAID areas of RAID Level 1 and a second fre- 
quency d^bution table is created for virtual bfocks 
associated with parity RAID areas of RAID Level 5. If ao 
three or more RAID Levels are used, then three a more 
con-esponding tat)les are created. An example use of this 
memory arrangement might be to ntigrate "middle-aged" 
virtual blocks to a level of storage that has performance 
characteristics most suitable for their particular data. 3s 

Rg. 9 shows two frequency distribution tabtes 60 
and 70. Lt^per table 60 is used to charact^ize virtual 
blocks that are associated with RAID Level 1 storage. 
Lower table 70 is t^ed to characterize virtual t)focks that 
are associated witti RAID Level 5 storage. 40 

Rg. 9 also Ulusbates the effect of migration on the 
frequency distributionlablesforthediflerent RAID levels. 
The method for using the table for migration is shown in 
the migration nrxxle of operation in Rg. 12. For purposes 
of th^exanrple, the tables in Rg. 9 are based upon the 45 
distrtxition criteria of data access recerK:y and the inter- 
vals are temporal in scope. Assume that the RAID man- 
agement system 16 wants to migrate "oMer" virtual btock 
from min-or to parity storage to free up mirror storage for 
new data. In this case, tiie mirror RAID areas operate as so 
the source RAID areas for data migration and the parity 
RAID areas operate as the target RAID areas (step 120 
in F^. 12). 

The RAID management system first divides ttie pop- 
ulation of virtual bfocks In table 60 into a plurality of equal ss 
size segments having approximately the same number 
of elements (step 122). R)r example, the RAID manage- 
ment system might divide the population into five seg- 
ments, as shown at the top of table 60 in Rg. 9. Ft>ra 



population of 566. each segment shoukl theoretically 
represent approximately 1 13virtual t)locks. Thefirstseg- 
ment spans intervals 1 -A because intervals 1 -3 only have 
1 05 virtual blocks, befow the threshokJ 113 virtual bfocte 
for each segment 

The RAID management system then chooses the 
first segment since it will contain the okJer virtual blocks 
(step 124 in Fig. 12). Any virtual block having a time 
stanp foiling within the temporal intervals referenced 
generically as 1-4 will satisfy this criterion. The RAID 
management system determines the uppermost tx>und- 
ary value of the first segment whk;h, in this example, is 
the ending boundary time value of interval 4. The ending 
boundary time value is computed by taking the beginning 
boundary time value of interval 4 and adding the size of 
interval 4. The erKiing boundary time value is used as a 
threshoM for comparison. 

The RAID management system then searches 
through the virtual blocks in the application-level virtual 
storage space to actually locate one or more virtoal 
bfocks (step 126). The time stannp of each virtoal block 
is compared to the threshoM time value to see if the vir- 
tual bfock has a time stanp earlier than the tiireshokJ 
time value, thereby placing the virtual bk>ck in the first 
segment Once a sufficient number of virtual bfocks are 
found, the search is halted. The located virtoal blocks are 
ttien migrated from minror storage to parity storage (step 
128). 

The two frequency distribution tables 60 and 70 are 
updated to reflect the migration (step 130). Suppose tiiat 
the migration of three virtual bfocks previously repre- 
sented in interval 2 of RAID Level 1 tatile 60 is now 
reflected in inten/al 10 of RAID Level 5 table 70. The 
count associated with interval 2 in table 60 is simply dec- 
remented by three (from 48 to 45) to reflect the loss of 
the virtual blocks. Similarly, the count associated witii 
interval 10 in table 70 is incremented by three (from 47 
to 50) to reflect ttie gain of the virtual blocks. It is noted 
that the time stamps of the three virtual blocks do not 
change during migration. Rattier, only the statistical 
infomfiation for each RAID level is modified to reflect the 
cfiange in distribution resulting from the data migration. 

It is noted tfiat the migration is preferat)ly performed 
in a continuous fashfon, as opposed to a t)atch proce- 
dure. Accordingly, once a aiterion-satisfying block is 
found, it is migrated, and tiie tables updated, in a block- 
t)y-block manner. 

The use of frequency distrit>ution tables affords more 
advantages than the toll sorting techniqua The statistical 
approach yields signifk^antiy better performance than 
sorting methods, while empfoying signifk;antly less 
memory resource. The process of segmenting the pop- 
ulation, choosing a segment to Identify virtual blocks, and 
ttien quickly searching to find ttie virtual t>locks approxi- 
mates the toll sorting technique. Only it accomplishes tiie 
task in a more efffolent and cost effective manner 
t)ecause it enables intelligent selection witfiout expense. 
Additionally, the process can be stopped as soon as a 
sufficient number off virtual bfocks are found in the foen- 
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tified segment By stopping earlier, the truncated time 
frame is generally significanHy less than that required to 
perform a full sort 

Another benefit is that the processing time is a linear 
function of the number of virtual blocks (for searching) 5 
unlike the non-linear function of full sorting techniques. 
Thus. oMeryyounger and less recentlyAnore recentiy 
accessed virtual blocks can be more quicMy identified 
with respect to tiie amount of user data stored on the 
disk array. io 

The use of statistical frequency disfribution tables 
can also be used to select data for movement between 
different types of storage devices. For example, if the 
data storage system was further equipped with a t)ack- 
up storage mechanism, such as reel-to-reel tape stor- is 
age, the statistical frequency distributkMi table couM be 
used to select '^ancienT storage blocks for data mcve- 
ment from the disk array to tiie back-up storage mecha- 
nism. 

In compliance with the statute, tiie invention has so 
been described in language more or 1^ specific as to 
structural and metfiodical features. It is to be understood, 
however, that the invention is not limited to the specif 
features shown and descrft>ed. since the means herein 
disclosed comprise preferred forms of putting the inven- 2S 
tion into effect. TTie invention is, therefore, claimed in any 
of its forms or modifications within the proper scope of 
the appended claims appropriately interpreted in accord- 
ance witti tiie doctrine off equivalents. 
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Claims 

1. A method for selecting data in a data storage sys- 
tem; the data storage system (10) comprising mul- 
tf>le storagedisks (1 2) ttiat define a physical storage 35 
space (34). the physk»l storage space b^ng 
mapped into a virtual storage space which presents 
tiie physical storage space as a population of multi- 
pie virtual btocks; the method comprising the follow- 
ing steps: 40 

aeating a bounded frequency distribution 
table (GO) that characterizes data stored on the data 
storage system (10) according to a predetermined 
distribution criteria, tiie frequency d^lMition tat)le 
(60) having a selected number of intervals wtiere 45 
individual intervals have various counts of the 
number of virtual bk)cks ttiat satisfy the distrlxjtion 
criteria for the associated interval; 

segmenting the frequency disfribution table 
(60) to kientify a selection criterion for virtual blocks so 
that are a subset of the population; and 

searching the virtual blocks in the virtual stor- 
age space to focate one or more virtual btocks that 
satisfy the selection criterion. 

55 

2. A method according to daim 1 further comprising 
the additfonal step of tagging tiie virtual blocks with 
a time stamp indicating when thedata was stored in 
the virtual btocks; and 



wherein tiie predetermined disfribution crite^ . 
ria is data access recency and the intervals are tem- 
poral in scope where the count associated with a 
temporal interval represents the number of virtual 
bfodcs having time stamps within tiiattemporal inter- 
val. 

3. A meUiod according to claim 1 further comprising 
ttie additional step of tagging the virtual t>locks witii 
a frequency stamp indicating the number of times 
the data ttierein has been accessed; and 

wherein the predetermined distribution crite- 
ria is data access frec^jency and the intervals relate 
to frequency values where the counts associated 
with the intervals represent the number of virtual 
k)locks having frequency values within the respective 
intervals. 

4. A metiiod accoiding to daim 1 further comprising 
the following additional steps: 

updating the frequency disbibutfon table (60) 
as new virtual t>kx:ks are used; and 

saki updating step alternately oomprising: (1) 
in the event that a new virtual t)lock satisfies tiie dis- 
fribution aiteriafor an existing interval, incrementing 
the count associated witii that existing interval; and 
(2) in the event tiiat a new virtual Uock fails to satisfy 
tiie distnlxition criteria for an existing interval, gen- 
erating a new interval and entering an initial count 
for ttiat new interval. 

5b A metiiod according to claim 5 wherein the step of 
generating a new interval comprises: 

joining at least ^o intervals within the fre- 
quency disbibution table (60) to define a joint inter- 
val, the count associated mUn the joint interval t)eing 
the sum of the counts of the two intervals tfiat are 
joined; and 

defining a new interval which extends the dis- 
tribution aiteria to cover the new virtual block. 

6. A method according to claim 1 furttier comprising 
tiie following additional steps: 

updating the frequency distrikxrtfon tat^e (60) 
as an existing virtual bkx:k is accessed; and 

saki updating step comprising deaementing 
the count of an interval for which the existing virtual 
block satisfied ttie distribution criteria prior to access 
and incrementing the count of andher interval for 
which the existing virtual block satisfied the distriftxj- 
tion criteria subsequent to access. 

7. A metiiod for selecting data for migration in a hier- 
archic data storage system (10); the data storage 
system comprising multiple storage disks (12) that 
define a physical storage space (34). tine physical 
storage space toeing mapped into a RAID-level vir- 
tual storage space (40) which presents tfie physk^l 
storage space as at least two RAID areas that store 
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data accoiding to drfferent RAID levels, the RAID-., 
level virtual storage space being mapped into an 
application-level virtual storage space (50) which 
presents the physical storage space as a population 
of multiple virtual blocks; the method comprising the 5 
following steps: 

creating a t)ounded first frequency distribu- 
tion table (60) that characterizes data stored in a first 
RAID area under a first RAID level according to a 
predetermined distrit>ution criteria, the first fre- 70 
quency distribution table (60) having a selected 
number of intervals, individual intervals having vari- 
ous counts of the number of virtual blocks that satisfy 
ttie distribution criteria for the associated interval; 

creating a tx>unded secorxl frequertcy distri- fs 
bution table (70) that cfiaracterizes data stored in a 
second RAID area under a second RAID level 
according to a predetermined distribution criteria, 
the second frequency distribution table (70) having 
a selected number of interaals, individual intervals 20 
having various counts of the numbers of virtual 
blocks that satisfy the distribution criteria for the 
associated interval; 

defining a plurality of segments of the first 
population, individual segments including one or ss 
more inten/als of the first frequency (jRstribution 
table; 

(loosing at least one segment from the plu- 
rality of segments; 

searching the virtual blocks in the applk;ation- 30 
level virtual storage space (50) to locate one or more 
virtual bkx^ that satisiy the distribub'on criteria for 
the one or more intervals included within the cfiosen 
segment, the located virtual t)locks representing 
data stored in thef irst RAID area under thef irst f^lD 35 
level; and 

migrating the located virtual bk)cks from the 
f iist RAID area to the second RAID area. 



store data according to various RAID levels; the 
applicatk>n-level virtual storage space (50) present- 
ing the physical storage space as a population of 
multiple virtual blocks: 

the RAID management system (1 6) migrating 
virtual blocks between tiie first and second RAID 
areas on the storage disks (12); 

first and second bounded frequency distritxj- 
tion tables (60, 70) that characterize data stored in 
a first and second RAID areas according to a prede- 
termined distribution aiteria, the frequency distribu- 
tion tables having selected numbers of intervals, 
individual intervals having various counts of the 
number of virtual bkxAs that satisfy the distribution 
criteria for the associated intervals; and 

the RAID management system (1 6) selecting 
the virtual blocks for migration by segmenting one of 
the first and second frequency distribution tables 
(60, 70) to identify a selection criterkm for virtual 
trfocks which comprise a sut>set of their respective 
populations. 

1 0. A data storage system according to claim 18 further 
comprising means for marking the virtual t)kx:ks with 
a stamp fiaving a monotonk:ally increasing value. 



8. A method accoiding to claim 9 further comprising 40 
the addittonal step of tagging the virtual blocks with 
a stamp having a monotonically inaeasing value so 
that newer stamped virtual bkxAs are counted in last 
inten/ate of tiie first and second frequency distribu- 
tion tat)les. 45 



9. A data storage system comprising: 

a disk anray (1 1) having a plurality of storage 
disks (12) ttiat define a physical storage space (34); 

adiskarray controller (14) coupled to the disk so 
anay (1 1) for coordinating data transfer to and from 
the disks (12); 

a RAID management system (16) operatively 
coMpled to tiie disk array controller (1 4) for mapping 
a RAlD-4evel virtual storage space (40) artd an appli- 55 
cation-level virtual storage space (50) into the phys- 
ical storage space (34) of the storage disks (12); tiie 
RAID-level virtual storc^e space (40) presenting the 
physical storage space as multiple RAID areas that 
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